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1 .  SCOPE 

1.1  Objective. 

This  Test  Operations  Procedure  (TOP)  describes  test  methods  and 
techniques  for  measuring  and  evaluating  the  technical  performance  of  a 
System  Under  Test  (SUT)  when  the  SUT  is  operated  to  and  beyond  speci¬ 
fications  in  order  to  determine  its  response  to  high  levels  of  stimuli 

1.2  Limitations . 


1.2.1  The  extremely  short  period  allowed  for  the  writing  of  this  TOP 
along  with  the  great  diversity  of  potential  systems  to  be  tested  has 
precluded  the  writing  of  a  document  which  conclusively  covers  the 
topic  of  stress  level  testing.  This  dociunent  should  be  considered  as 
a  TOP  which  outlines  the  basic  requirements  and  the  basic  test  metho¬ 
dology  for  conducting  stress  level  testing. 

1.2.2  This  TOP  includes  only  the  electronics  aspects  of  stress  level 
testing.  It  does  not  consider  the  classical  environmental  and  mechan 
ical  stress  testing  of  materials;  these  types  of  stress  testing  are 
covered  in  MIL-STD-810D. 

2.0  FACILITIES  AND  INSTRUMENTATION 

Listed  below  are  the  general  types  of  instrumentation/facilities 
required  for  stress  testing  of  C3I  systems. 
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2.1  Facilities. 

Applicable  facility  requirements  include: 

2.1.1  A  computer  facility  capable  of  simulating  and  modeling  functions 
of  the  item  being  tested  both  stand  alone  and  within  a  complex  electro¬ 
magnetic  environment  (both  communications  and  electronic  warfare). 

2.1.2  A  facility  capable  of  providing  repeatable  message  stimuli 
directly,  (either  by  direct  connection  or  by  using  item  under  test  - 
communications)  to  the  system  being  tested.  This  facility  should  be 
scenario  driven,  should  offer  real  time  operation  and  should  provide 
data  collection,  reduction  and  analysis  capabilities.  A  facility  of 
this  type  is  described  in  Appendix  C. 

2.1.3  A  facility  capable  of  providing  repeatable  Radio  Frequency  (RF) 
stimuli  to  the  item  under  test.  This  stimuli  should  represent  realis¬ 
tic  RF  signal  effects  to  be  found  in  selected  military  situations  (or 
as  stated  in  requirements)  and  should  be  provided  to  the  system  being 
tested  using  digital,  direct  coupled  RF,  close  coupled  RF  or  low 
powered  RF  techniques.  This  facility  should  be  scenario  driven, 
should  offer  real  time  operation  and  should  provide  data  collection 
capabilities.  A  facility  of  this  type  is  descrilsed  in  Appendix  D. 

2.1.4  A  range  facility  providing  the  following. 

2. 1.4.1  Adequate  range  space  to  deploy  segments  of  selected  military 
communications  structure. 

2. 1.4. 2  Controlled  air  space  or  traffic  free  air  space. 

2.1.4. 3  Test  control  station. 

2. 1.4. 4  Space  position  traclcing  and  plotting  capability. 

2 . 1 . 4 . 5  Range  timing . 

2 . 1 . 4 . 6  Meteorological  support . 

2. 1.4. 7  Secure  Range  communications. 

2.1.5  If  the  SUT  is  an  avionics  system  which  is  not  part  of  an 
integrated  airfreune  system,  the  appropriate  J-series  aircraft  plat- 
form(s)  may  be  required. 

2.2  Instrumentation. 

In  addition  to  the  instrumentation  embedded  in  the  facilities 
mentioned  above  the  following  items  are  required  on  an  as  needed 
basis. 

2.2.1  RF  Communications  Simulator(s) . 

2.2.2  RF  Noncommunications  Simulator ( s ) . 


TOP  6-1-002 


30  August  1985 

2.2.3  RF  Januner  ( s ) . 

2.2.4  Range  RF  Monitoring  Equipment ( s ) . 

2.3  Characteristics/Requirements . 

The  characteristics  of  the  instrumentation/ facilities  are  deter¬ 
mined  by  the  specific  requirements  of  the  equipment(s)  and  system(s) 
being  tested.  Instrumentation/ facilities  shall  have  the  capability, 
both. qualitatively  and  quantitatively,  to  produce  stimuli-  which  drive 
the  system  being  tested  into  failure.  Where  applicable,  instrumen¬ 
tation  (to  include  that  embedded  in  facilities)  shall  be  calibrated  by 
approved  methods  and  instrumentation  traceable  to  the  National  Bureau 
of  Standards.  The  facilities  and  instrumentation  indicated  above  can 
provide  the  necessary  characteristics  and  setups  required  to  perform 
subtests  indicated  in  this  TOP. 

3.0  REQUIRED  TEST  CONDITIONS 

The  test  officer  should,  during  the  test  process: 

3.1  Review  TECOM  Pamphlet  70-3,  Research,  Development,  and  Acquisition 
and  Project  Engineer's  Handbook  for  guidance  on  test  planning,  execu¬ 
tion,  reporting  and  post- test  activities. 

3.2  Establish  and/or  continually  maintain  a  readily  accessible 
project  log  and  project  file. 

3.3  Review  local  installation  project  officer's  handbook,  standard 
operating  procedures  (SOPs)  and  implementing  directives  which  govern 
the  administrative  processes  for  preparing  test  plans,  conducting 
tests,  preparing  reports,  reporting  to  the  Test  Resources  Management 
System  (TRMS)  and  budgeting. 

3.4  Acquire  and  review  all  descriptive,  instructional  and  specifi¬ 
cation  materiel  (to  include  software  documentation)  on  the  test  items 
issued  by  the  government  and  contractor ( s )  for  checking  the  test  plan 
subtest  objectives,  criteria,  facility  and  instrumentation  requirements 

3.5  Acquire  the  necessary  information  to  complete  the  required 
simulation/modeling  before  the  arrival  of  the  system  under  test. 

3.6  Ensure  availability  of  appropriate  facilities  and  coordinate  the 
test  support  requirements  including  qualified  personnel,  equipment, 
maintenance,  spare  parts  and  instrumentation. 

3.7  Review  the  detailed  test  plan  and  coordinate  any  changes  to  the 
detailed  test  plan  resulting  from  the  simulation/modeling  effort. 

3.8  Record  as  a  minimum,  the  following  data. 

3.8.1  Nomenclature,  serial  number (s),  manufacturer's  name,  and 
function  of  the  item(s)  under  test.  Completely  describe  version  of 
test  item  software  being  used  or  simulated. 


TOP  6-1-002 


30  August  1985 


3.8.2  Nomenclature,  serial  number,  accuracy  tolerance,  calibration 
requirements  and  latest  calibration  date  o£  test  equipment  selected 
for  the  tests. 

3.8.3  Damage  to  test  item(s)  incurred  during  transit  and/or  manufac¬ 
turing  defects.  Record  missing/incomplete  software  as  well  as  software 
being  modified  at  the  time  of  testing. 

3.8.4  Test  item  photograph. 

3.8.5  Establish  test  system  measures  of  performance  (MOP);  identify 
those  MOP  that  are  potentially  stress  limited.  For  the  purpose  of 
this  document  the  term  MOP  is  taken  to  mean  those  technical  and 
operational  characteristics  which  must  be  evaluated  to  determine/cali¬ 
brate  a  system's  effectiveness  and  utility. 

3.8.6  Identify  the  specification  limitations  of  each  MOP.  Determine 
inputs  that  will  stress  each  MOP  beyond  specified  point. 

3.8.7  Identify  and  document  instrumentation  or  measurement  error 
acceptable  for  recording  data  relative  to  each  MOP  condition.  Deter¬ 
mine  required  scunple  size  for  each  MOP  measurement. 

3.8.8  Check  with  the  responsible  frequency  management  agency  to 
ensure  authorization  for  radiation  in  the  required  frequency  bands 
during  the  anticipated  test  period  if  a  field  test  is  to  be  performed. 

3.8.9  Orient  all  test  personnel  on  the  test  objectives,  test  require¬ 
ments,  schedule  of  events,  and  safety  precautions. 

3.8.10  Have  on  hand  the  survey  of  non-ionizing  radiation  hazards  by 
the  U.S.  Army  Environmental  Hygiene  Agency  and  adhere  to  all  precau¬ 
tions  contained  therein. 

3.8.11  If  required,  assure  that  the  Airworthiness  Release  for  the  SUT 
has  been  received  from  AVSCOM. 

3.8.12  Organize  test  team  and  establish  responsibilities  for  test 
conduct,  reporting  and  data  control. 

3.8.13  Check  to  ensure  that  test  equipment  and  accessories  are 
available,  operational  and  meet  certified  calibration  requirements. 

3.8.14  Perform  an  operational  check  of  the  test  item(s)  to  ensure 
normal,  correct  functioning. 

3.8.15  Prepare  adequate  safety  precautions  to  provide  safety  for 
personnel  and  equipment  and  ensure  that  all  safety  SOPs  are  observed 
throughout  the  test. 

3.8.16  Prepare  and  monitor  a  data  acquisition  plan  sufficient  to 
ensure  that  enough  samples  of  all  measurements  are  taken  to  provide 
statistical  confidence  of  final  data  in  accordance  with  TOP  3-1-002. 
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3.8.17  Identify  specific  interfaces  required  between  the  instrumen¬ 
tation  and  the  system  under  test. 

3.8.18  Identify  requirement(s)  for  interactive  capabilities. 

4.0  PERFORMANCE  TEST  PROCEDURES 

The  generic  procedures  to  be  followed  for  stress  testing  are 
given  in  this  section.  More  specific  requirements/procedures  for 
-  Intercept  and  e*2  systems- are -given-in  Appendixes  A  and  B  respectively. 

4.1  General. 


4.1.1  Stress  testing  is  defined  as  the  exercising  of  a  system  to  and 
beyond  its  design  specifications  in  order  to  determine  its  response  to 
high  levels  of  stimuli;  it  is  testing  to  the  point  at  which  a  system 
saturates  as  well  as  the  determination  of  the  effects  of  such  satur¬ 
ation  on  system  operation.  In  the  context  of  electronic  systems  it  is 
especially  important  to  know  where  and  how  (i.e.  graceful  or  catas¬ 
trophic)  a  system  fails  since  this  information  will  enable  the  eventual 
system  user  to  design  for  gradual  degradation  or  to  plan  missions 
around  the  failure  point;  this  information  will  allow  the  user  to 
avoid  "red  line  conditions"  during  equipment  usage. 

4.1.2  Stress  Level  Testing  is  no  more  than  an  extension  of  typical 
development  testing.  Instead  of  testing  only  to  the  specified  limits 
of  each  MOP,  testing  is  extended  by  adding  additional  stimuli  until 
the  test  item  fails,  either  gracefully  or  catastrophically,  to  perform 
the  function  being  considered.  The  point  and  type  of  failure  is  noted 
for  future  use  and  the  test  is  rerun  some  predetermined  number  of 
times  to  verify  its  accuracy/statistical  significance.  The  details  of 
the  level  of  stress  developed  shall  be  documented  for  future  tests  of 
similar  items  on  a  product  improvement  version  of  the  test  item. 

4.1.3  A  generalized  method  of  Stress  Level  Testing  for  electronic 
systems  has  been  developed  at  the  Electronic  Proving  Ground.  This 
method  consists  of  testing  in  a  simulation/modeling  environment, 
testing  in  a  bench/closed  loop  environment  and  testing  in  a  controlled 
field  test  environment.  These  three  different  phases  of  the  test 
process  and  their  relationships  to  each  other  are  shown  in  figure  1. 
These  relationships  allow  for  the  synergistic  interaction  of  the  test 
phases  and  lead  to  a  logical,  economical  coupling  of  field,  bench  and 
simulation  testing. 

4.1.4  The  specific  details  of  stress  load  testing  are  completely 
dependent  on  the  type  and  in  many  cases  the  specific  system  being 
tested.  The  test  details  listed  following  are,  hence,  very  general  in 
nature  and  are  meant  to  show  general  concepts  rather  than  specific 
cookbook  ideas.  To  lend  some  detail  to  the  discussion  two  system 
types  are  briefly  discussed  with  regards  to  the  test  process.  These 
types,  an  intercept  system  and  a  command  and  analysis  system  are 
discussed  respectively  in  appendixes  A  and  B. 


Figure  1.  Generalized  Test  Methodology 
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4.2.1  After  acquiring  and  reviewing  all  available/necessary  documen¬ 
tation,  the  MOPs  relating  to  the  item  being  tested  will  be  determined. 
The  specification  requirements /limits  of  each  MOP  will  be  described 
and  those  MOPs  which  appear  stress  dependent  will  be  identified.  An 
excunple  of  a  stress  dependent  MOP  would  be  an  MOP  which  says  that  a 
receiver  can  handle  up  to  20  signals  at  one  time  without  any  degrad¬ 
ation  in  quality.  In  that  case  it  would  be  important  to  determine 
what  would  happen  to  that  receiver  when  faced  with  21,  22,  .  .  .  .  n 
signals. 


4.2.2  Once  applicable  MOPs  have  been  decided  the  envirohmen€  in  which 
the  item/system  is  designed  to  work  will  be  identified.  This  environ¬ 
ment  will  serve  as  input  to  future  testing. 

4.3  Simulation/Modelinq  Testing. 

Early  in  the  test  process  the  system  under  test  will  be  described 
in  a  model  and  inserted  in  a  tactical  model  of  its  planned  deployment 
environment.  This  simulation  will  be  used  to  provide  increasingly 
heavy  stress  loads  to  the  system  under  test  and  will  identify  those 
MOPs  that  require  empirical  testing.  For  RF  dependent  SUTs  the  entire 
RF  environment  is  initially  considered  and,  through  the  elimination  of 
signals  that  do  not  affect  the  system  under  test,  is  ultimately  scaled 
down  to  a  manageable  level  for  closed  loop  testing. 

4.4  Bench/Closed  Loop  Testing. 

4.4.1  Subsequent  to  receipt  of  answers  from  the  simulation/- 
modeling  phase  of  testing  the  Bench/Closed  loop  portion  of  the  test 
plan  is  written/updated  and  the  system  under  test  hardware  is  received. 
In  this  series  of  tests  the  system  is  immersed  in  a  realistic  (data  or 
RF)  environment  through  either  direct  connection  or  close  coupling 
with  devices  producing  environmental  stimuli.  This  close  connection 
between  the  environment  and  the  system  being  tested  allows  the  SUT  to 
be  operated  in  a  simulated,  realistic,  controlled  environment  and 
enables  the  SUT  to  be  stressed  way  beyond  any  capability  that  could 
exist  and  be  duplicated  in  a  controlled  field  test  environment. 

4.4.2  Closed  loop  testing  for  message  (data)  systems  requires  a 
device  to  transfer  messages  to/from  the  system  under  test.  An  item  of 
this  type  has  been  developed  at  EPG.  This  item,  the  Test  Item  Stimu¬ 
lator,  generates  and  records  a  data  environment  which  exercises  and 
stresses  systems  under  test.  The  TIS  provides  controlled  and  repeatable 
technical/operational  digital  messages  representative  of  multiple 
C3I/IEW  systems  to  the  SUT  and  monitors  and  records  the  SUT  responses 

in  real  time.  Appendix  C  contains  a  detailed  description  of  the  TIS. 

4.4.3  Closed  loop  testing  for  RF  systems  requires  a  device  to  simulate 
at  RF  frequencies  the  signals,  both  friendly  and  unfriendly,  found  in 

a  battlefield  environment.  An  item  of  this  type  is  currently  being 
developed  by  EPG.  This  item,  the  Stress  Loading  Facility  (SLF),  has 
the  capability  to  test  large  systems  in  a  controlled  environment.  It 
will  mitigate  the  risk  of  security  disclosures,  will  provide  signal 
loading  to  and  beyond  system  design  points  and  will  enable  testing  to 
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be  done  free  of  contaminating  signals.  The  SLF  will  provide  controlled 
and  repeatable  technical  and  operational  simulated  signals  to  items 
being  tested  and  will  monitor  and  record  the  SUT  response  to  signals 
in  real  time.  Appendix  D  contains  a  detailed  description  of  the  SLF. 

4.5  Field  Testing. 

Subsequent  to  the  completion  of  closed  loop  testing  the  field 
test  phase  of  the  test  plan  is  written/updated  and  a  minimum  sized 
field  test  is  designed.  In  this  series  of  tests  the  SUT  is  immersed 
in  a  realistic,  field-  environment  relative  to  the  MOP*being"r:onsidered. 
The  SUT  is  operated  either  by  troops  or  technical  personnel  to  answer 
specific  system  related  questions  as  derived  from  previous  closed  loop 
and  simulationZ-modeling  testing.  This  testing  besides  resolving  open 
questions  will  verify  the  accuracy  of  previous  testing,  will  verify 
the  adequacy  of  the  system  to  work  in  a  field  environment  and  will 
provide  feedback  to  the  closed  loop  and  simulation  efforts. 

5 . 0  DATA  PRESENTATION 

5 . 1  General. 

Data  will  be  reduced  from  its  raw  state  into  some  predetermined 
form  and  analyzed  as  defined  by  the  test  plan.  Data  from  each  of  the 
three  test  phases  will  be  processed  to  preserve  commonality  and  to  be 
used  for  correlation.  Data  reduction  and  analysis  techniques  used 
will  include  but  not  be  limited  to  the  following. 

5.1.1  Data  will  be  identified  and  time  tagged  for  correlation. 

5.1.2  Data  will  be  organized  into  tabular  or  graphic  form. 

5.1.3  Data  presented  will  be  sufficient  to  support  the  conclusion( s ) 
drawn . 

5.1.4  Data  collected  will  be  retained  in  machine  readable  form  until 
all  test  issues  have  been  resolved. 

5 . 2  Simulation/Modelinq  Tests. 

Data  will  be  acquired,  reduced  and  analyzed  as  described  in  the 
test  plan.  There  is  no  standard  data  format  that  covers  the  different 
types  of  systems  being  stressed. 

5.3  Bench/Closed  Loop  Testing. 

Data  will  be  acquired,  reduced  and  analyzed  as  described  in  the 
test  plan.  There  is  no  standard  data  format  that  covers  the  different 
types  of  systems  being  stressed. 

5.4  Field  Testing. 

Data  will  be  acquired,  reduced  and  analyzed  as  described  in  the 
test  plan.  There  is  no  standard  data  format  that  covers  the  different 
types  of  systems  being  stressed. 
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APPENDIX  A 

STRESS  LEVEL  TESTING  OF  INTERCEPT  SYSTEMS 


A. 1  General. 


A. 1.1  This  category  of  testing  (except  for  the  vulnerability 
testing  of  intelligence  equipments)  deals  primarily  with  systems 
which  intercept  and  analyze  foreign  (red)  signals.  It  is  assumed 
that  any  message  traffic  passed  between  the  SUT  and  any  other 
friendly  automated  system  will  be  tested  as  a  command  and  control 
systems  test  and  that  vulnerability  testing  of  any  communications 
linlcs  will  be  tested  as  in  a  vulnerability  test  using  the  DVAL 
methodology. 

A. 1.2  Testing  of  intercept  systems  requires  a  large  (in  terms  of 
numbers  of  signals)  representative  environment.  Since  the 
acquisition  of  actual  or  replica  target  emitters  in  the  )cinds  and 
numbers  required  by  the  SUT  is  impractical,  the  environment  will 
be  provided  by  a  set  of  RF  thre^*.  simulator  devices. 

A. 1.3  During  a  typical  test  the  SUT  will  be  operated  in  both 
closed  loop  and  controlled  field  test  environments.  During 
closed  loop  testing  special  technical  scenarios  will  be  used  to 
provide  RF  signals  (typically  into  the  receiver  input  ports)  of 
the  SUT.  It  is  expected  that  closed  loop  testing  will  answer 
many  test  questions  and  will  allow  the  environment  required  for 
controlled  field  testing  to  be  downsized.  Controlled  field 
testing,  as  shown  in  figure  A-1,  will  deploy  simulators  (either 
red  or  blue)  at  representative  distances  from  the  SUT.  These 
simulators  will  produce  many  simultaneous /near  simultaneous 
signals.  They  will  be  synchronized  through  a  command  lin)c  to  a 
central  processor  using  a  master  scenario.  Data  reduction  will 
be  accomplished  on  both  a  quic)c  loolc  and  post-test  basis.  For 
the  testing  of  an  airborne  system,  these  methods  will  be 
augmented  by  the  ability  to  trade  multiple  (typically  three) 
aircraft  and  a  means  for  instrumenting  the  SUT  aboard  the  aircraft. 

A. 2  MOP  Development. 

A  list  of  selected  MOPs  for  an  intercept  system  are  given 
below.  Those  items  deemed  to  relate  to  stress  testing  are  mar)ced 
by  an  asteris)c  prior  to  the  listing. 

*  .Rate  of  Processing  the  Field  of  View 

*■  .Out  of  Field  of  View  Processing,  Rejection  or 

Rejection/Processing  of  Out  of  Field  of  View  signals. 

.Emitter  Characteristic  (Time  Frequency)  Measurement  Accuracy 
with  Attention  to  Multipath/Field  Effects 

*  .Saturation/Graceful  Degradation 

*  .Saturation/ Amount  of  Data  Lost 

*  .False/Erroneous  Data  Handling  Capability 
.Repairability 

*  .File  Modulation  Capability 

Update 
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* 

* 

* 

* 

* 


* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 


Enter  Data 
Record  Data 

.Command  and  Control  Links  Reliability,  Error  Rate, 
Susceptibility,  Vulnerability 
.Pre-Test  Inventory  and  Inspection 
.Environmental  Testing 
.System  Reliability 
.System  Availability 
.System  Maintainability 
.System  Compatibility 
.  Interoperability 
.Vulnerability  to  ECM 
.Vulnerability  to  ESM 
.Human  Factors  Engineering 
.MANPRINT 
.Safety 
.Bite  Testing 
.Logistics  Support 
.Software  Evaluation 

.Documentation  and  Training  Evaluation 
.Probability  of  Intercept 
.Probability  of  False  Alarm 

.Processing  Time  of  Intercept  (How  long  from  intercept  to 
information  output) 

.Digital  Modulation  Capability  (FSK,  PPM) 

.  Encrypt ic/Nonencryptic  Voice  and  Data  Capability 

.Digital  Encryptic  Recovery  Capability 

.Spread  Spectrum  Capability 

•Time  and  Frequency  Multiplexing  Capability 

.AM,  FM,  SSB,  CW  Capability 

.Voice/Digital  Recording  Capability 

•Recording  Time/Signal  with  Attention  to  Multipath  and  Field 
Effects 

•Co-channel  Interference 
•System  Setup  and  Initialization  Time 
•Tear  Down  Time 
•Site  Preparation 

•  Mobility/Transportability 
•Mast  stability 

•  ECCM 


A. 3  Specific  Test  Requirements. 

A. 3.1  The  applicable  MOPs  will  be  considered  in  the  test  design 
process  for  each  of  the  three  phases  of  testing.  For  the  purposes 
of  brevity  this  document  will  discuss  only  the  MOP  "Processing 
time  of  Intercept." 

A. 3. 2  Upon  definition  of  MOP  requirements  a  simulation  of  the 
SUT/SUT  environment  will  be  developed.  This  simulation  (if 
technically  feasible)  will  be  tailored  to,  in  this  case,  answer 
the  processing  time  MOP.  Signals  will  be  provided  to  the  SUT 
incrementally  up  to  and  beyond  the  specified  requirements  level 
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and  the  intercept  time  monitored  to  determine  when,  where  and  how 
the  system  fails. 

A. 3. 3  Subsequent  to  the  simulation  effort  the  SUT  will  be  run  in 
a  closed  loop  situation  using  the  Stress  Loading  Facility.  The 
exact  SLF  connections /type  and  numbers  of  signals  provided  relate 
to  the  SUT  and  its  corresponding  environment.  It  is  envisioned 
that  the  environment  of  figure  A-1  will  be  created  using  the  SLF 
as  simulator /monitor  equivalents.  Again  signals  will  be  provided 
incrementally  to  the  SUT  up  to  and  beyond  its  specified  require¬ 
ments  level  and  the  results  of  the  test  used  to  verify  simulation 
results  (if  a  simulation  test  was  feasible)  and  determine  empiri¬ 
cally  where  and  how  the  system  fails. 

A. 3. 4  Subsequent  to  simulation/closed  loop  testing  a  limited 
field  test  will  be  run.  Testing  will  use  the  environment  type 
shown  in  figure  A-1  and  will  be  configured  to  both  validate 
simulation/closed  loop  test  results  as  well  as  to  specifically 
answer  the  MOP  being  considered. 


30  August  1985 


TOP  6-1-002 


APPENDIX  B 

STRESS  LEVEL  TESTING  OF  COMMAND  AND  CONTROL  SYSTEMS 
B.  1  Conunand  and  Control  Systems  Testing. 

B.1.1  This  category  (except  for  vulnerability  testing  of  C2 
equipments)  deals  primarily  with  the  transfer  of  data  between  two 
or  more  friendly  (blue)  equipments  or  systems.  This  data  transfer 
is  done  either  via  tactical  or  strategic  circuits  or  by  communi¬ 
cations  equipment  indigenous  to  the  SUT. 

B.1.2  Since  future  systems  of  this  type  will  be  primarily 
digital,  computer-based  systems,  it  is  probable  that  data  passed 
to/from  the  SUT  will  be  in  the  form  of  digital  messages.  In 
order  to  test  the  ability  of  the  SUT  to  process  this  traffic,  the 
SUT  must  be  connected  to  the  system(s)  with  which  it  is  designed 
to  interface  or  to  a  message  driver  or  stimulator  which  can 
emulate  such  systems.  The  number  of  interfacing  systems  is  quite 
large  for  some  SUTs.  These  interfacing  systems  most  often  are 
not  economically  available  for  a  test,  and  they  are  not  amenable 
to  the  strict  control  and  repeatability  necessary  for  development 
testing.  Therefore  a  programmable  device  which  can  represent  an 
interfacing  system  and  provide  controlled,  repeatable  stimuli  is 
the  preferred  approach. 

B,1.3  During  a  typical  test,  the  stimulator (s)  will  be  connected 
as  shown  in  figure  B-1  at  one  or  more  points  to  the  SUT.  The 
stimulators,  connected  to  a  common  control  networJc,  will  pass 
scenario-driven,  realistic,  time-correlated  data  to  the  SUT. 

Data  received  from  the  SUT  (to  incltzde  messages  received,  time  of 
messages  received,  etc.)  will  be  recorded  in  the  stimulator  for 
reduction  (both  in  real-time  and  batch)  for  analysis  and  reporting. 

B.2  MOP  Development. 

A  list  of  selected  MOPs  for  a  command  and  analysis  system 
are  given  below.  Those  items  deemed  to  relate  to  stress  testing 
are  marJced  by  an  asterislc  prior  to  the  listing. 

*  .Time  of  Response  (Time  from  Request /Command  from  an  Operator 
to  Time  of  Aclcnowledgement/Display/Response) 

-  Minimum  Response  Time  (Data  re  AcJcnowledgement  of  Receipt) 
to  any  Request. 

-  Time  for  Display  of  Command  Menus,  Report  Formats, 

Wor)csheet  Formats  and  Similar  Fixed  Data,  Bloc)cs 

-  Time  for  Display  of  Directives  to  Messages /Reports  Stores 
in  the  system  to  Include  WorJc  Queues  of  Feeder  Inputs  to 
be  Processed 

-  Time  for  Display/Listing  of  Historical  or  Long-term 
Messages/Reports  Stored  in  the  System 

-  Time  for  Displays  of  Selective  Correlation/Sorts  of  stored 
Data 


B-1 


Figure  B-1.  Command  and  Control  System  Test  Concept 
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.Capability  to  Process  Cornpartmented/Noncompartmented  Material 

*  .Automatic  Message  Routing  Capability 

-  Priority  Routing 

-  Originator  Routing 

-  Message  Type  Routing 

.Text  Editing  Capability 

-  Paging  and  Scrolling 

-  Display  Content  Transfer 

-  Graphics  Capability 

-  Protected  and  Open  Area  Capability  and  Modification 

-  Mass  Storage  Capability 

-  Short-term  Storage  Capability 

-  Storage  Transfer  Capability 

-  Mass  Storage  Emergency  Destruction  Capability 

*  . Communication  Links 

-  Message  Assembly  Capability 

-  Error  Correction  Capability 

-  Message  Acknowledgements/Retransmission  Capability 

-  Code  Transliteration  Capability 

-  Communication  Line  Terminations  Characteristics 

-  Secure  Communication  Capability 

-  Variable  Link  Capability  (HP,  VHP,  UHP,  VOICE /TTY/ DATA) 

*  .Interrupts  of  System  Utilization  by  Diagno«bic  Routines 

*  .Number  of  Reports /Day  Capability 

*  .Number  of  Characters /Day  Capability 

*  .Number  of  Interfaces  Capability 

*  .Calculation  Capability 

*  .System  Overload/Graceful  Degradation 
.Pre-Test  Inspection  and  Inventory 

.Environmental  Testing  (Operation  in  Extreme  Heat,  Cold,  Salt, 
Pog,  Etc.) 

.Reliability 

.Availability 

. Maintainability 

.Compatibility 
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*  .Interoperability 
. Susceptibility 

*  .Vulnerability 

*  .Vulnerability  to  ESM 

*  .Human  Factors  Engineering 

*  .MANPRINT 
•Safety 

.BITE  Testing  (System  Diagnostic  to  Lowest  Field  Replaceable 
Units) 

.Logistic  Support  Package 
.Software  Evaluation,  Maintained>ility 
•Documentation  and  Training 
•Setup/Tear  down  time 
•Site  Preparation 
•Transportability  of  System 
.Mobility  of  System 
.Mast  Stability 

•Hearability  as  a  Function  of  Antenna  Height 
B. 3  Specific  Test  Requirements. 

B.3.1  The  applicable  MOPs  will  be  considered  in  the  test  design 
process  for  each  of  the  three  phasee  of  -teating-  For  the  purposes 
of  brevity  this  document  will  discuss  only  the  MOP  "Time  of 
Response . " 

B.3.2  Upon  definition  of  MOP  requirements  a  simulation  of  the 
Data  environment  will  be  developed.  This  simulation  will  be 
tailored,  in  this  case,  to  answer  the  time  of  response  MOP. 
Messages  will  be  provided  to  the  SUT  incrementally.„.up  to  and 
beyond  the  specified  requirements  level  and  the  response  time 
monitored  to  determine  when,  where  and  how  the  system  fails. 

B.3.3  Subsequent  to  the  simulation  effort  the  SUT  will  be  run  in 
a  closed  loop  situation  using  the  Test  Item  Stimulator.  The 
exact  TIS  connections/types  and  numbers  of  messages  provided 
relate  to  the  SUT  and  its  corresponding  data  environment.  It  is 
envisioned  that  the  data  environment  of  figure  B-1  will  be 
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created  using  the  TIS  to  provide  external  messages  to  the  SUT. 
Again  a  message  will  be  provided  incrementally  to  the  SUT  up  to 
and  beyond  its  specified  requirements  level  and  the  results  of 
the  test  used  to  verify  simulation  results  (if  a  simulation  was 
feasible)  and  determine  empirically  where  and  how  the  system 
fails. 


B.3.4  Subsequent  to  simulation/closed  loop  testing  a  limited 
field  test  will  be  run.  Testing  will  use  the  environment  type 
shown  in  Figure  B-1  and  will  be  configured  to  both  validate 
simulation/closed  loop  test  results  as  well  as  to  specifically 
answer  the  MOP  being  considered. 
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APPENDIX  C 

TEST  ITEM  STIMULATOR  (TIS)  DESCRIPTION 


1.  The  TIS  Is  a  transportable  or  fixed  site  test  system  for  testing 
C3I  systems.  The  TIS  generates  the  data  environment  to  exercise  and 
stress  the  systems  under  test  to  measure  their  interoperability, 
software,  and  system  performance.  The  TIS  provides  controlled  and 
repeatable  digital  messages  for  stimulation  of  multiple  C3I  systems 
under  test  and  monitors/records  their  responses  in  real-time.  The 
data  environment  generation  is  performed  by  a  single  or  multi-proces¬ 
sor  system  under  control  of  both  prespecified  and  real-time  inter¬ 
active  scenarios.  It  utilizes  state-of-the-art  commercial  computers, 
displays,  licensed  commercial  software,  and  government  owned/con¬ 
trolled  applications  software.  Either  military  or  commercial  communi¬ 
cations  may  be  used  to  link  the  TIS  with  systems  under  test. 

2.  The  TIS  supplies  a  data  environment  which: 

*  Complies  with  message  standards  and  specifications. 

*  May  also  include  carefully  selected  and  controlled  errors. 

*  Does  not  include  spurious  or  unintentional  messages. 

*  Allows  test  messages  to  be  automatically  associated,  singly  or 
in  sets,  with  functional  test  objectives. 

*  Allows  message  timing  to  be  specified  and  synchronized  with 
other  tests  or  test  control  events. 

*  Provides  large  volumes  of  validated  input  messages  to  stress 
test  large  systems  with  confidence. 

*  Measures  communications  and  processing  throughputs. 

*  Can  be  designed  to  independently  stress  system 
functions /resources  to  avoid  intracteUsle  test  results. 

3.  Description  of  how  the  TIS  functions: 

a.  The  TIS  initially  generates  a  simulated  data  environment  which 
it  presents  to  the  SUT  via  military  or  commercial  data  links.  The  TIS 
then  records  system  responses  in  real-time  to  ascertain  the  validity 
of  interface  designs,  compliance  with  design  standards,  and  the 
performance  resulting  from  achieved  interoperability. 

b.  Each  TIS  can  simultaneously  supply  message  traffic  on  28 
narrow-band  and  4  JTIDS  data  links  or  the  equivalent  data  rate  in 
almost  any  combination.  The  TIS  monitors /records  output  messages  in 
real-time  and  records  data  for  subsequent  off-line  analysis. 
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c.  The  TIS  is  transportable  and  capable  of  operating  in  a  wide 
range  of  geographic  and  climatic  conditions. 

d.  The  TIS,  as  a  stand-alone  test  support  system,  supports  test 
operations  during  pre-test,  real-time,  and  post-test  phases.  During 
pre-test,  the  software  provided  assists  test  planners  in  scenario 
creation,  message  preparation,  critical  data  identification,  and  in 
conducting  pre-test  diagnostics.  Following  the  test,  the  software 
aids  evaluation  by  providing  utilities  for  data  reduction,  collation, 
and  consolidation;  identifying  significant  events;  providing  a  library 
of  analytic  support  modules;  providing  ’’friendly"  interactive  man-ma¬ 
chine  interface  of  graphic  and  textual  data  terminals. 

e.  When  operation  in  the  role  of  a  test  driver,  the  TIS: 

(1)  Stimulates  tactical  activities  in  the  systems  under 
test/participating  systems  by  simulating  tactical  sensor  or  intelli¬ 
gence  sources. 

(2)  Emulates  the  message  streeuns  and  communications  logic  of 
C3I  systems  and  interfaces  with  the  system  under  test  to  support 
interoperability  testing. 

(3)  Encodes  and  transmits  scenario  driven  messages  in  the 
correct  format  and  content  while  using  the  appropriate  communications 
protocols . 

(4)  Receives,  decodes,  displays,  and  records  message  streams 
received  from  the  system  under  test  as  a  result  of  their  processing, 
operator  activity,  or  battle  management  in  real-time. 

4.  FUNCTIONAL  DESCRIPTION.  The  following  functional  capabilities  are 
provided  by  the  TIS; 

a.  Scenario  generation  and  modification 

b.  Library  of  tactical  system-specific  appliques  (SSA)  (software 
packages)  including;  TADIL-J,  MCS,  TACFIRE,  AN/TSQ-73,  HAWK,  and 
ASAS. 

c.  Time-metered  scenario  control  driven  by  actual  exercise  time 
(IRIG-B) . 


d.  Ability  to  stop,  start,  and  modify  the  test  in  real-time. 

e.  Time  tagged  (IRIG-B)  data  collection  of  test  messages. 

f.  Quick-look  analysis  of  test  data  for  test  control. 

g.  Library  of  data  reduction  and  analysis  software  for  post-test 
evaluation. 


h.  Interactive  graphic  display  subsystem 
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APPENDIX  D 

STRESS  LOADING  FACILITY  (SLF)  DESCRIPTION 


1.  Stress  Loading  Facility  (SLF)  is  an  integrated  testing  system  to 
achieve  a  dense  electromagnetic  threat  test  environment  while  simul¬ 
taneously  monitoring  key  performance  parameters.  The  SLF  will  provide 
for  test  scenario  generation,  automated  generation  of  accurate  commu- 
nications/non-communications  RF  signal  signatures  in  dense  scenarios 
(stress  load),  automated  test  data  collection  with  time  code  correla¬ 
tion,  semi -automated  test  data  reduction  and  analysis,  centralized 
computer  coordination  of  all  test  events  (including  message  stimu¬ 
lation),  test  control,  and  test  repeatability  in  support  of  C3I 
systems  development  testing.  This  will  be  accomplished  in  a  closed 
environment  to  exclude  unwanted  signals  from  contaminating  the  test 
results  and  to  prevent  the  radiation  of  classified  or  sensitive 
signals  into  the  external  environment  during  testing. 

2.  The  potential  role  of  the  SLF  is  to  provide  a  capability  between 
the  computer  simulation/modeling  and  open  range  controlled  field 
testing,  in  which  a  controlled  and  repeatable  test  environment  can  be 
applied  to  the  system  under  test  which  could  not  practically  be  done 
in  a  controlled  open  range  field  test. 

3.  The  SLF  is  comprised  of  the  following  subsystems  as  shown  on  the 
attached  block  diagram  (Figure  D-1); 

*  TEST  CONTROL  STATION  (TCS) 

*  CENTRAL  COMPUTER  SUBSYSTEM  (CCS) 

*  COMMUNICATIONS  THREAT  SIMULATOR  (CTS) 

*  NON-COMMUNICATIONS  THREAT  SIMULATOR  (NCTS) 

*  FUNCTIONAL  SYSTEMS  SIMULATOR  (FSS) 

*  TEST  DATA  MONITORING  SUBSYSTEM  (TDMS) 

4.  Descriptions  of  how  the  SLF  functions: 

a.  The  SLF  generates  a  simulated  RF  environment  based  on  a  time 
event  scenario  that  is  developed  during  the  pre-test  period.  This  RF 
environment  also  includes  the  associated  modulations  and  types  of 
information  that  would  be  transmitted  by  this  RF  environment.  The 
systems/functions/components  that  are  associated  with  the  system  under 
test  to  allow  it  to  operate  realistically,  and  are  not  physically 
available  for  the  test,  will  be  simulated  within  the  SLF. 

b.  The  CTS,  NCTS,  FSS,  and  TDMS  each  have  their  own  dedicated 
control  processor  which  will  allow  them  to  operate  in  a  stand-alone 
mode  without  the  SLF  CCS  in  unique  situations.  Normally,  in  system 
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tests  the  entire  SLF  would  function  under  the  control  of  the  CCS  and 
associated  TCS. 

c.  Subsystem  functions  are: 

(1)  TEST  CONTROL  STATION 

*  CONTROL  AND  MANAGEMENT  OF  REAL-TIME  OPERATIONS 

*  TIMING  INFORMATION  ACQUISITION 

*  AUDIO  INTERFACE  WITH  ALL  SLF  SUBSYSTEMS 

*  OPERATOR  INTERFACE  FOR  OFF-LINE  OPERATIONS  FOR  TEST 
PLANNING,  SCENARIO  DEVELOPMENT  AND  EMITTER  PARAMETER  FILE 
MAINTENANCE 

(2)  CENTRAL  COMPUTER  SUBSYSTEM 

*  INTEGRATED  TEST  CONTROL 

*  DATA  REDUCTION  AND  ANALYSIS  FOR  BOTH  QUICK-LOOK  AND  POST 
TEST  REQUIREMENTS 

*  UPDATING  AND  MANAGEMENT  OF  DATA  BASE(S)  CONTAINING  THE 
PRE-TEST  CALIBRATION  AND  REAL-TIME  TEST  DATA 

*  GENERATION  OF  THE  INTEGRATED  SCENARIO  FOR  THE  ENTIRE  TEST 
TO  BE  RUN  IN  THE  SLF 

*  GENERATION  OF  THE  INDIVIDUAL  SIMULATOR  SUBSYSTEM 
SCENARIOS  AND  ASSOCIATED  SCENARIO  FILES 

*  CREATION,  UPDATING,  AND  MAINTENANCE  OF  EMITTER  PARAMETER 
FILES 

*  CREATION,  UPDATING,  AND  MAINTENANCE  OF  DATA  MESSAGE  FILES 

*  SLF  SOFTWARE  MANAGEMENT 

(3)  COMMUNICATIONS  THREAT  SIMULATOR 

*  GENERATE  VOICE  AND  NON-VOICE  COMMUNICATIONS  SIGNALS,  0.5 
TO  500  MHz 

*  CONTROLS  ASSOCIATED  SOFTWARE  USING  INTERNAL  PROCESSOR 

*  RUNS  PREDESIGNATED  SCENARIOS 

*  AUTOMATIC  CONTROL  OF  MODULATION,  RF  SOURCES,  AND  RF 
DISTRIBUTION 

*  SCENARIO  GENERATION  CAPABILITY  FOR  STAND  ALONE  MODE 
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(4)  NON- COMMUNICATIONS  THREAT  SIMULATOR 

*  GENERATE  PULSED  RADAR,  NAVIGATION,  AND  OTHER 
NON-COMMUNICATIONS  SIGNALS,  0.5  GHz  TO  18.0  GHz 

*  CONTROLS  ASSOCIATED  SOFTWARE  USING  INTERNAL  PROCESSOR 

*  RUNS  PREDESIGNATED  SCENARIOS 

*  AUTOMATIC  CONTROL  OF  MODULATION,  RF  SOURCES,  AND  RF 
DISTRIBUTION 

*  SCENARIO  GENERATION  CAPABILITY  FOR  STAND  ALONE  MODE 

(5)  FUNCTIONAL  SYSTEMS  SIMULATOR 

*  SIMULATES  EXTERNAL  FUNCTIONS  THAT  NEED  TO  BE  ACTIVATED, 
STIMULATED,  CONTROLLED,  OR  CHECKED  FOR  STATUS 

*  SIMULATION  OF  SUPPORT/CONTROL  SYSTEMS  TO  THE  SUT 

*  SIMULATION  OF  C3I  SYSTEMS  THAT  THE  SUT  WOULD  NORMALLY 
INTERFACE/ INTEROPERATE  WITH 

*  SIMULATION  OF  ADDITIONAL  UNITS  OF  THE  SUT  NOT  AVAILABLE 

*  SUT  EQUIPMENT/SYSTEM  STATUS  MONITORING 

*  CONTROLS  ASSOCIATED  SOFTWARE  USING  INTERNAL  PROCESSOR 

*  RUNS  PREDESIGNATED  SCENARIOS 

*  SCENARIO  GENERATION  CAPABILITY 

(6)  TEST  DATA  MONITORING  SUBSYSTEM 

*  PERFORMS  PRE-TEST  CALIBRATION 

*  PERFORMS  PRE-TEST  READINESS  CHECKS 

*  PERFORMS  ENVIRONMENT  MONITORING 

*  PERFORMS  SUT  PARAMETER  MONITORING 

*  PERFORMS  POST  TEST  MEASUREMENTS  (DRIFT) 

*  PERFORMS  REAL-TIME  QUICKLOOK  DATA 

d.  The  RF  distribution  designs  for  the  CTS  and  NCTS  are  generic 
as  far  as  possible.  However,  each  system  to  be  tested  will  have  to  be 
looked  at  in  detail  to  determine  the  system  specific  interface  applique 
that  will  be  required.  The  same  approach  applies  to  the  FSS. 
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AM  Amplitude  Modulation 
AVSCOM  Aviation  Systems  Command 
BITE  Built  In  Test  Equipment 
C2  Command  and  Control 
CW  Continous  Wave 

C3I  Command,  Control,  Communications  and  Intelligence 

DT  Development  Test 

ECM  Electronic  Countermeasures 

ESM  Electronic  Support  Measures 

EWI  Electronic  Warfare  Intelligence 

FM  Frequency  Modulation 

FSK  Frequency  Shift  Keying 

HF  High  Frequency 

lEW  Intelligence  Electronic  Warfare  . 

-LOB  Line  of  Bearing 

MOP  Measures  of  Performance 
OB  Order  of  Battle 

RF  Radio  Frequency 

SLF  Stress  Loading  Facility 
SOP  Standard  Operating  Procedures 
SSB  Single  Side  Band 

SUT  System  Under  Test 

TECOM  Test  and  Evaluation  Command 
TIS  Test  Item  Stimulator 

TOP  Test  Operations  Procedure 

TRMS  Test  Resources  Management  System 
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TTY  Teletype 
UHF  Ultra  High  Frequency 

USAEP6  United  States  Army  Electronic  Proving  Ground 
VHF  Very  High  Frequency 


